This invention relates to arrangements for detecting the presence, type and position of occupants in vehicles and objects exterior of vehicles, e.g., in a driver""s blind spot, primarily using optics.
1. Prior Art on Out of position occupants and rear facing child seats
Whereas thousands of lives have been saved by airbags, a large number of people have also been injured, some seriously, by the deploying airbag, and over 100 people have now been killed. Thus, significant improvements need to be made to airbag systems. As discussed in detail in U.S. Pat. No. 5,653,462 reference above, for a variety of reasons vehicle occupants may be too close to the airbag before it deploys and can be seriously injured or killed as a result of the deployment thereof. Also, a child in a rear facing child seat that is placed on the right front passenger seat is in danger of being seriously injured if the passenger airbag deploys. For these reasons and, as firstly publicly disclosed in Breed, D. S. xe2x80x9cHow Airbags Workxe2x80x9d presented at the International Conference on Seatbelts and Airbags in 1993, in Canada, occupant position sensing and rear facing child seat detection systems are required.
Initially, these systems will solve the out-of-position occupant and the rear facing child seat problems related to current airbag systems and prevent unneeded airbag deployments when a front seat is unoccupied. However, airbags are now under development to protect rear seat occupants in vehicle crashes and all occupants in side impacts. A system will therefore be needed to detect the presence of occupants, determine if they are out-of-position and to identify the presence of a rear facing child seat in the rear seat. Further automobiles are expected to have eight or more airbags as protection is sought for rear seat occupants and from side impacts. In addition to eliminating the disturbance and possible harm of unnecessary airbag deployments, the cost of replacing these airbags will be excessive if they all deploy in an accident needlessly.
Inflators now exist which will adjust the amount of gas flowing to the airbag to account for the size and position of the occupant and for the severity of the accident. The vehicle identification and monitoring system (VIMS) discussed in U.S. Pat. No. 5,829,782 will control such inflators based on the presence and position of vehicle occupants or of a rear facing child seat. As discussed more fully below, the instant invention is an improvement on that VIMs system and uses an advanced optical system comprising one or more CCD (charge coupled device) or CMOS arrays and particularly active pixel arrays plus a source of illumination preferably combined with a trained neural network pattern recognition system.
Others have observed the need for an occupant out-of-position sensor and several methods have been disclosed in U.S. patents for determining the position of an occupant of a motor vehicle. Each of these systems, however, has significant limitations. For example, in White et al. (U.S. Pat. No. 5,071,160), a single acoustic sensor and detector is described and, as illustrated, is mounted lower than the steering wheel. White et al. correctly perceive that such a sensor could be defeated, and the airbag falsely deployed, by an occupant adjusting the control knobs on the radio and thus they suggest the use of a plurality of such sensors.
Mattes et al. (U.S. Pat. No. 5,118,134) described a variety of methods of measuring the change in position of an occupant including ultrasonic, active or passive infrared and microwave radar sensors, and an electric eye. The sensors measure the change in position of an occupant during a crash and use that information to access the severity of the crash and thereby decide whether or not to deploy the airbag. They are thus using the occupant motion as a crash sensor. No mention is made of determining the out-of-position status of the occupant or of any of the other features of occupant monitoring as disclosed in one or more of the above cross-referenced patents and patent applications. It is interesting to note that nowhere does Mattes et al. discuss how to use active or passive infrared to determine the position of the occupant. As points out in one or more of the above cross-referenced patents and patent applications, direct occupant position measurement based on passive infrared is probably not possible and, until very recently, was very difficult and expensive with active infrared requiring the modulation of an expensive GaAs infrared laser. Since there is no mentioned of these problems, the method of use contemplated by Mattes et al. must be similar to the electric eye concept where position is measured indirectly as they occupant passes by a plurality of longitudinally spaced-apart sensors.
The object of an occupant out-of-position sensor is to determine the location of the head and/or chest of the vehicle occupant relative to the airbag since it is the impact of either the head or chest with the deploying airbag which can result in serious injuries. Both White et al. and Mattes et al. describe only lower mounting locations and their sensors in front of the occupant such as on the dashboard or below the steering wheel. Both such mounting locations are particularly prone to detection errors due to positioning of the occupant""s hands, arms and legs. This would require at least three, and preferably more, such sensors and detectors and an appropriate logic circuitry which ignores readings from some sensors if such readings are inconsistent with others, for the case, for example, where the driver""s arms are the closest objects to two of the sensors.
White et al. also describe the use of error correction circuitry, without defining or illustrating the circuitry, to differentiate between the velocity of one of the occupant""s hands as in the case where he/she is adjusting the knob on the radio and the remainder of the occupant. Three ultrasonic sensors of the type disclosed by White et al. might, in some cases, accomplish this differentiation if two of them indicated that the occupant was not moving while the third was indicating that he or she was. Such a combination, however, would not differentiate between an occupant with both hands and arms in the path of the ultrasonic transmitter at such a location that they were blocking a substantial view of the occupant""s head or chest. Since the sizes and driving positions of occupants are extremely varied, it is now believed that pattern recognition systems and preferably trained pattern recognition systems, such as neural networks, are required when a clear view of the occupant, unimpeded by his/her extremities, cannot be guaranteed.
Fujita et al., in U.S. Pat. No. 5,074,583, describe another method of determining the position of the occupant but do not use this information to suppress deployment if the occupant is out-of-position. In fact, the closer the occupant gets to the airbag, the faster the inflation rate of the airbag is according to the Fujita et al. patent, which thereby increases the possibility of injuring the occupant. Fujita et al. do not measure the occupant directly but instead determine his or her position indirectly from measurement of the seat position and the vertical size of the occupant relative to the seat (occupant height). This occupant height is determined using an ultrasonic displacement sensor mounted directly above the occupant""s head.
As discussed above, the optical systems described herein are also applicable for many other sensing applications both inside and outside of the vehicle compartment such as for sensing crashes before they occur as described in patent application Ser. No. 08/239,978 cross-referenced above, for a smart headlight adjustment system and for a blind spot mirror (also described in U.S. provisional patent application Ser. No. 60/202,424).
2. Definitions
The use of pattern recognition is central to the instant invention as well as to one or more of those disclosed in the cross-referenced patents and patent applications above. xe2x80x9cPattern recongitionxe2x80x9d as used herein will generally mean any system which processes a signal that is generated by an object, or is modified by interacting with an object, in order to determine which one of a set of classes that the object belongs to. Such a system might determine only that the object is or is not a member of one specified class, or it might attempt to assign the object to one of a larger set of specified classes, or find that it is not a member of any of the classes in the set. The signals processed are generally electrical signals coming from transducers which are sensitive to either acoustic or electromagnetic radiation and, or electromagnetic, they can be either visible light, infrared, ultraviolet or radar or low frequency radiation as used in capacitive sensing systems.
A trainable or a trained pattern recognition system as used herein means a pattern recognition system which is taught various patterns by subjecting the system to a variety of examples. The most successful such system is the neural network. Not all pattern recognition systems are trained systems and not all trained systems are neural networks. Other pattern recognition systems are based on fuzzy logic, sensor fusion, Kalman filters, correlation as well as linear and non-linear regression. Still other pattern recognition systems are hybrids of more than one system such as neural-fuzzy systems.
To xe2x80x9cidentifyxe2x80x9d as used herein will usually mean to determine that the object belongs to a particular set or class. The class may be one containing, for example, all rear facing child seats, one containing all human occupants, or all human occupants not sitting in a rear facing child seat depending on the purpose of the system. In the case where a particular person is to be recognized, the set or class will contain only a single element, i.e., the person to be recognized.
To xe2x80x9cascertain the identity ofxe2x80x9d as used herein with reference to an object will generally mean to determine the type or nature of the object (obtain information as to what the object is), i.e., that the object is an adult, an occupied rear facing child seat, an occupied front facing child seat, an unoccupied rear facing child seat, an unoccupied front facing child seat, a child, a dog, a bag of groceries, etc.
An xe2x80x9coccupying itemxe2x80x9d or xe2x80x9coccupantxe2x80x9d of a seat or xe2x80x9cobjectxe2x80x9d in a seat may be a living occupant such as a human being or a dog, another living organism such as a plant, or an inanimate object such as a box or bag of groceries.
In the description herein on anticipatory sensing, the term xe2x80x9capproachingxe2x80x9d when used in connection with the mention of an object or vehicle approaching another will usually mean the relative motion of the object toward the vehicle having the anticipatory sensor system. Thus, in a side impact with a tree, the tree will be considered as approaching the side of the vehicle and impacting the vehicle. In other words, the coordinate system used in general will be a coordinate system residing in the target vehicle. The xe2x80x9ctargetxe2x80x9d vehicle is the vehicle that is being impacted. This convention permits a general description to cover all the cases such as where (i) a moving vehicle impacts into the side of a stationary vehicle, (ii) where both vehicles are moving when they impact, or (iii) where a vehicle is moving sideways into a stationary vehicle, tree or wall.
xe2x80x9cOut-of-positionxe2x80x9d as used for an occupant will generally means that the occupant, either the driver or a passenger, is sufficiently close to an occupant protection apparatus (airbag) prior to deployment that he or she is likely to be more seriously injured by the deployment event itself than by the accident. It may also mean that the occupant is not positioned appropriately in order to attain the beneficial, restraining effects of the deployment of the airbag. As for the occupant being too close to the airbag, this typically occurs when the occupant""s head or chest is closer than some distance such as about 5 inches from the deployment door of the airbag module. The actual distance value where airbag deployment should be suppressed depends on the design of the airbag module and is typically farther for the passenger airbag than for the driver airbag.
3. Patter recognition prior art
Japanese Patent No. 3-423337 (A) to Ueno discloses a device for detecting the driving condition of a vehicle driver comprising a light emitter for irradiating the face of the driver and a means for picking up the image of the drive and storing it for later analysis. Means are provided for locating the eyes of the driver and then the irises of the eyes and then determining if the driver is looking to the side or sleeping. Ueno determines the state of the eyes of the occupant rather than determining the location of the eyes relative to the other parts of the vehicle passenger compartment. Such a system can be defeated if the driver is wearing glasses, particularly sunglasses, or another optical device which obstructs a clear view of his/her eyes. Pattern recognition technologies such as neural networks are not used.
U.S. Pat. No. 5,008,946 to Ando uses a complicated set of rules to isolate the eyes and mouth of a driver and uses this information to permit the driver to control the radio, for example, or other systems within the vehicle by moving his eyes and/or mouth. Ando uses natural light and illuminates only the head of the driver. He also makes no use of trainable pattern recognition systems such as neural networks, nor is there any attempt to identify the contents of the vehicle nor of their location relative to the vehicle passenger compartment. Rather, Ando is limited to control the vehicle devices by responding to motion of the driver""s mouth and eyes.
U.S. Pat. No. 5,298,732 to Chen also concentrates on locating the eyes of the driver so as to position a light filter between a light source such as the sun or the lights of an oncoming vehicle, and the driver""s eyes. Chen does not explain in detail how the eyes are located but does supply a calibration system whereby the driver can adjust the filter so that it is at the proper position relative to his or her eyes. Chen references the use of automatic equipment for determining the location of the eyes but does not describe how this equipment works. In any event, there is no mention of illumination of the occupant, monitoring the position of the occupant, other that the eyes, determining the position of the eyes relative to the passenger compartment, or identifying any other object in the vehicle other than the driver""s eyes. Also, there is no mention of the use of a trainable pattern recognition system.
U.S. Pat. No. 5,305,012 to Faris also describes a system for reducing the glare from the headlights of an oncoming vehicle. Faris locates the eyes of the occupant utilizing two spaced apart infrared cameras using passive infrared radiation from the eyes of the driver. Again, Faris is only interested in locating the driver""s eyes relative to the sun or oncoming headlights and does not identify or monitor the occupant or locate the occupant relative to the passenger compartment or the airbag. Also, Faris does not use trainable pattern recognition techniques such as neural networks. Faris, in fact, does not even say how the eyes of the occupant are located but refers the reader to a book entitled Robot Vision (1991) by Berthold Horn, published by MIT press, Cambridge, Mass. A review of this book did not appear to provide the answer to this question. Also, Faris uses the passive infrared radiation rather than illuminating the occupant with active infrared radiation or in general electromagnetic radiation.
The use of neural networks as the pattern recognition technology is central to several of the implementations of this invention since it makes the monitoring system robust, reliable and practical. The resulting algorithm created by the neural network program is usually only a few dozen lines of code written in the C or C++ computer language as opposed to typically hundreds of lines when the techniques of the above patent to Ando, Chen and Faris are implemented. As a result, the resulting systems are easy to implement at a low cost, making them practical for automotive applications. The cost of the CCD and CMOS arrays, for example, have been prohibitively expensive until recently, rendering their use for VIMS impractical. Similarly, the implementation of the techniques of the above referenced patents requires expensive microprocessors while the implementation with neural networks and similar trainable pattern recognition technologies permits the use of low cost microprocessors typically costing less than $10 in large quantities.
The present invention preferably uses sophisticated trainable pattern recognition capabilities such as neural networks. Usually the data is preprocessed, as discussed below, using various feature extraction techniques. An example of such a pattern recognition system using neural networks on sonar signals is discussed in two papers by Gorman R. P. and Sejnowski, T. J. xe2x80x9cAnalysis of Hidden Units in a Layered Network Trained to Classify Sonar Targetsxe2x80x9d, Neural Networks, Vol. 1. pp. 75-89, 1988, and xe2x80x9cLearned Classification of Sonar Targets Using a Massively Parallel Networkxe2x80x9d, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36, No. 7, July 1988. Examples of feature extraction techniques can be found in U.S. Pat. No. 4,906,940 entitled xe2x80x9cProcess and Apparatus for the Automatic Detection and Extraction of Features in Images and Displaysxe2x80x9d to Green et al. Examples of other more advanced and efficient pattern recognition techniques can be found in U.S. Pat. No. 5,390,136 entitled xe2x80x9cArtificial Neuron and Method of Using Same and U.S. patent application Ser. No. 08/076,601 entitled xe2x80x9cNeural Network and Method of Using Samexe2x80x9d to S. T. Wang. Other examples include U.S. Pat. Nos. 5,235,339 (Morrison et al.), 5,214,744 (Schweizer et al), 5,181,254 (Schweizer et al), and 4,881,270 (Knecht et al). All of the above references are included herein by reference.
4. Optics
Optics can be used in several configuration for monitoring the interior of a passenger compartment of an automobile. In one known method, a laser optical system uses a GaAs infrared laser beam to momentarily illuminate an object, occupant or child seat, in the manner as described and illustrated in FIG. 8 of U.S. Pat. No. 5,829,782 cross-referenced above. The receiver can be a charge-coupled device or CCD (a type of TV camera), to receive the reflected light. The laser can either be used in a scanning mode, or, through the use of a lens, a cone of light can be created which covers a large portion of the object. In these configurations, the light can be accurately controlled to only illuminate particular positions of interest within the vehicle. In the scanning mode, the receiver need only comprise a single or a few active elements while in the case of the cone of light, an array of active elements is needed. The laser system has one additional significant advantage in that the distance to the illuminated object can be determined as disclosed in the ""462 patent.
In a simpler case, light generated by a non-coherent light emitting diode device is used to illuminate the desired area. In this case, the area covered is not as accurately controlled and a larger CCD or CMOS array is required. Recently, however, the cost of CCD and CMOS arrays has dropped substantially with the result that this configuration is now the most cost-effective system for monitoring the passenger compartment as long as the distance from the transmitter to the objects is not needed. If this distance is required, then the laser system, a stereographic system, a focusing system, a combined ultrasonic and optic system, or a multiple CCD or CMOS array system as described herein is required.
A mechanical focusing system, such as used on some camera systems can determine the initial position of an occupant but is too slow to monitor his/her position during a crash. A distance measuring system based on focusing is described in U.S. Pat. No. 5,193,124 (Subbarao) which can either be used with a mechanical focusing system or with two cameras, the latter of which would be fast enough. Although the Subbarao patent provides a good discussion of the camera focusing art and is therefore incorporated herein by reference, it is a more complicated system than is needed for the practicing of the instant invention. In fact, a neural network can also be trained to perform the distance determination based on the two images taken with different camera settings or from two adjacent CCD""s and lens having different properties as the cameras disclosed in Subbarao making this technique practical for the purposes of the instant invention. Distance can also be determined by the system disclosed in U.S. Pat. No. 5,003,166 (Girod) by the spreading or defocusing of a pattern of structured light projected onto the object of interest. Distance can also be measured by using time of flight measurements of the electromagnetic waves or by multiple CCD or CMOS arrays as is a principle teaching of this invention.
In each of these cases, regardless of the distance measurement system used, a trained pattern recognition system, as defined above, is used in the instant invention to identify and classify, and in some cases to located, the illuminated object and its constituent parts.
5. Optics and acoustics
The laser systems described above are expensive due to the requirement that they be modulated at a high frequency if the distance from the airbag to the occupant, for example, needs to be measured.
Both laser and non-laser optical systems in general are good at determining the location of objects within the two dimensional plane of the image and a pulsed laser radar system in the scanning mode can determine the distance of each part of the image from the receiver by measuring the time of flight through range gating techniques. It is also possible to determine distance with the non-laser system by focusing as discussed above, or stereographically if two spaced apart receivers are used and, in some cases the mere location in the field of view can be used to estimate the position relative to the airbag, for example. Finally, a recently developed pulsed quantum well diode laser also provides inexpensive distance measurements as discussed below.
Acoustic systems are additionally quite effective at distance measurements since the relatively low speed of sound permits simple electronic circuits to be designed and minimal microprocessor capability is required. If a coordinate system is used where the z axis is from the transducers to the occupant, acoustics are good at measuring z dimensions while simple optical systems using a single CCD are good at measuring x and y dimensions. The combination of acoustics and optics, therefore, permits all three measurements to be made from one location with low cost components as discussed in commonly assigned U.S. Pat. Nos. 5,845,000 and 5,835,613 cross-referenced above.
One example of a system using these ideas is an optical system which floods the passenger seat with infrared light coupled with a lens and CCD or CMOS array which receives and displays the reflected light and an analog to digital converter (ADC), or frame grabber, which digitizes the output of the CCD or CMOS and feeds it to an Artificial Neural Network (ANN) and other pattern recognition system for analysis. This system uses an ultrasonic transmitter and receiver for measuring the distances to the objects located in the passenger seat. The receiving transducer feeds its data into an ADC and from there the converted data is directed into the ANN. The same ANN can be used for both systems thereby providing full three-dimension data for the ANN to analyze. This system, using low cost components, will permit accurate identification and distance measurements not possible by either system acting alone. If a phased array system is added to the acoustic part of the system, the optical part can determine the location of the driver""s ears, for example, and the phased array can direct a narrow beam to the location and determine the distance to the occupant""s ears.
Although the use of ultrasound for distance measurement has many advantages, it also has some drawbacks. First, the speed of sound limits the rate at which the position of the occupant can be updated to approximately 10 milliseconds, which though sufficient for most cases, is marginal if the position of the occupant is to be tracked during a vehicle crash. Second, ultrasound waves are diffracted by changes in air density that can occur when the heater or air conditioner is operated or when there is a high-speed flow of air past the transducer. Third, the resolution of ultrasound is limited by its wavelength and by the transducers, which are high Q tuned devices. Typically, the resolution of ultrasound is one the order of about 2 to 3 inches. Finally, the fields from ultrasonic transducers are difficult to control so that reflections from unwanted objects or surfaces add noise to the data.
6. Applications
The applications for this technology are numerous as described in the copending patents and patent applications listed above. They include (i) the monitoring of the occupant for safety purposes to prevent airbag deployment induced injuries, (ii) the locating of the eyes of the occupant (driver) to permit automatic adjustment of the rear view mirror(s), (iii) the location of the seat to place the occupant""s eyes at the proper position to eliminate the parallax in a heads-up display in night vision systems (iv) the location of the ears of the occupant for optimum adjustment of the entertainment system, (v) the identification of the occupant for security reasons, (vi) the determination of obstructions in the path of a closing door or window, (vii) the determination of the position of the occupant""s shoulder so that the seat belt anchorage point can be adjusted for the best protection of the occupant, (viii) the determination of the position of the rear of the occupants head so that the headrest can be adjusted to minimize whiplash injuries in rear impacts, (ix) anticipatory crash sensing, (x) blind spot detection, (xi) smart headlight dimmers, (xii) sunlight and headlight glare reduction and many others. In fact, over forty products alone have been identified based on the ability to identify and monitor objects and parts thereof in the passenger compartment of an automobile truck.
7. Other Prior Art
European Patent Application No. 98110617.2 (Publication No. 0 885 782 A1), corresponding to U.S. patent application Ser. No. 08/872,836 filed Jun. 11, 1997, describes a purportedly novel motor vehicle control system including a pair of cameras which operatively produce first and second images of a passenger area. A distance processor determines the distances that a plurality of features in the first and second images are from the cameras based on the amount that each feature is shifted between the first and second images. An analyzer processes the determined distances and determines the size of an object on the seat. Additional analysis of the distance also may determine movement of the object and the rate of movement. The distance information also can be used to recognize predefined patterns in the images and this identify objects. An air bag controller utilizes the determined object characteristics in controlling deployment of the air bag.
A paper entitled xe2x80x9cSensing Automobile Occupant Position with Optical Triangualtionxe2x80x9d by W. Chappelle, Sensors, December 1995, describes the use of optical triangulation techniques for determining the presence and position of people or rear-facing infant seats in the passenger compartment of a vehicle in order to guarantee the safe deployment of an air bag. The paper describes a system called the xe2x80x9cTakata Safety Shield xe2x80x9d which purportedly makes high-speed distance measurements from the point of air bag deployment using a modulated infrared beam projected from an LED source. Two detectors are provided, each consisting of an imaging lens and a position-sensing detector.
A paper entitled xe2x80x9cAn Interior Compartment Protection System based on Motion Detection Using CMOS Imagersxe2x80x9d by S. B. Park et al., 1998 IEEE International Conference on Intelligent Vehicles, describes a purportedly novel image processing system based on a CMOS image sensor installed at the car roof for interior compartment monitoring including theft prevention and object recognition. One disclosed camera system is based on a CMOS image sensor and a near infrared (NIR) light emitting diode (LED) array.
A paper entitled xe2x80x9cA 256xc3x97256 CMOS Brightness Adaptive Imaging Array with Column-Parallel Digital Outputxe2x80x9d by C. Sodini et al., 1988 IEEE International Conference on Intelligent Vehicles, describes a CMOS image sensor for intelligent transportation system applications such as adaptive cruise control and traffic monitoring. Among the purposed novelties is the use of a technique for increasing the dynamic range in a CMOS imager by a factor of approximately 20, which technique is based on a previously described technique for CCD imagers.
A paper entitled xe2x80x9cIntelligent System for Video Monitoring of Vehicle Cockpitxe2x80x9d by S. Boverie et al., SAE Technical Paper Series No. 980613, Feb. 23-26, 1998, describes the installation of an optical retina sensor in the vehicle and several uses of this sensor. Possible users are said to include observation of the driver""s face (eyelid movement) and the driver""s attitude to allow analysis of the driver""s vigilance level and warn him/her about critical situations and observation of the front passenger seat to allow the determination of the presence of somebody or something located on the seat and to value the volumetric occupancy of the passenger for the purpose of optimizing the operating conditions for air bags.
Ishikawa et al. (U.S. Pat. No. 4,625,329) describes an image analyzer (M5 in FIG. 1) for analyzing the position of driver inclusion an infrared light source which illuminates the driver""s face and an image detector which receives light from the driver""s face, determines the position of facial feature, e.g., the eyes in three dimensions, and thus determines the position of the driver in three dimensions. A pattern recognition process is used to determine the position of the facial features and entails converting the pixels forming the image to either black or white based on intensity and conducting an analysis based on the white area in order to find the largest contiguous white area and the center point thereof. Based on the location of the center point of the largest contiguous white area, the driver""s height is derived and a heads up display is adjusted so information is within the driver""s field of view. The pattern recognition process can be applied to detect the eyes, mouth, or nose of the driver based on the differentiation between the white and black areas.
Ando (U.S. Pat. No. 5,008,946) describes a system which recognizes an image and specifically ascertains the position of the pupils and mouth of the occupant to enable movement of the pupils and mouth to control electrical devices installed in the automobile. The system includes a camera which takes a picture of the occupant and applies algorithms based on pattern recognition techniques to analyze the picture, converted into an electrical signal, to determine the position of certain portions of the image, namely the pupils and mouth.
Masamori (U.S. Pat. No. 5,227,784) describes a system which is based on radar, specifically it is a collision avoidance system aimed at detecting vehicles which are at some distance from the vehicle.
Suzuki et al. (U.S. Pat. No. 50,26,153) describes a vehicle tracking control for continuously detecting the distance and direction to a preceding vehicle irrespective of background dark/light distribution. In this system, every vehicle must have a light on its rear that emits a constant or time varying signal and two photoelectric sensors that zero in on the light emitted from the preceding vehicle are used and thereby determine both the distance and angular position of the preceding vehicle.
Krumm (U.S. Pat. No. 5,983,147) describes a system for determining the occupancy of a passenger compartment including a pair of cameras mounted so as to obtain binocular stereo images of the same location in the passenger compartment. A representation of the output from the cameras is compared to stored representations of known occupants and occupancy situations to determine which stored representation the output from the cameras most closely approximates. The stored representations include that of the presence or absence of a person or an infant seat in the front passenger seat.
Farmer et al. (U.S. Pat. No. 6,005,958) describes a method and system for detecting the type and position of a vehicle occupant utilizing a single camera unit. The single camera unit is positioned at the driver or passenger side A-pillar in order to generate data of the front seating area of the vehicle. The type and position of the occupant is used to optimize the efficiency and safety in controlling deployment of an occupant protection device such as an air bag.
Principle objects and advantages of the optical sensing system in accordance with the invention are:
1. To recognize the presence of a human on a particular seat of a motor vehicle and to use this information to affect the operation of another vehicle system such as the airbag, heating and air conditioning, or entertainment systems, among others.
2. To recognize the presence of a human on a particular seat of a motor vehicle and then to determine his/her position and to use this position information to affect the operation of another vehicle system.
3. To determine the position, velocity or size of an occupant in a motor vehicle and to utilize this information to control the rage of gas generation, or the amount of gas generated by an airbag inflator system.
4. To determine the presence or position of rear seated occupants in the vehicle and to use this information to affect the operation of a rear seat protection airbag for frontal, side and/or rear impacts.
5. To recognize the presence of a rear facing child seat on a particular seat of a motor vehicle and to use this information to affect the operation of another vehicle system such as the airbag system.
6. To determine the approximate location of the eyes of a driver and to use that information to control the position of one or more of the rear view mirrors of the vehicle.
7. To monitor the position of the head of the vehicle driver and determine whether the driver is falling asleep or otherwise impaired and likely to lose control of the vehicle and to use that information to affect another vehicle system.
8. To provide an occupant position sensor which reliably permits, and in a timely manner, a determination to be made that the occupant is out-of-position, or will become out-of-position, and likely to be injured by a deploying airbag and to then output a signal to suppress the deployment of the airbag.
9. To provide an anticipatory sensor that permits accurate identification of the about-to-impact object in the presence of snow and/or fog whereby the sensor is located within the vehicle.
10. To provide a smart headlight dimmer system which sense the headlights from an oncoming vehicle or the tail lights of a vehicle in front of the subject vehicle and identifies these lights differentiating them from reflections from signs or the road surface and then sends a signal to dim the headlights.
11. To provide a blind spot detector which detects and categories an object in the driver""s blind spot or other location in the vicinity of the vehicle, and warns the driver in the event the driver begins to change lanes, for example, or continuously informs the driver of the state of occupancy of the blind spot.
12. To provide a occupant position determination in a sufficiently short time that the position of an occupant can be tracked during a vehicle crash.
13. To provide an occupant vehicle interior monitoring system which is not affected by temperature or thermal gradients.
14. To provide an occupant vehicle interior monitoring system which has high resolution to improve system accuracy and permits the location of body parts of the occupant to be determined.
15. To provide an occupant vehicle interior monitoring system which reduces the glare from sunlight and headlights by imposing a filter between the eyes of an occupant and the light source.
16. To provide a camera system for interior and exterior monitoring, which can adjust on a pixel by pixel basis for the intensity of the received light.
These and other objects and advantages will become apparent from the following description of the preferred embodiments of the vehicle identification and monitoring system of this invention.
Briefly though, in order to achieve at least one of the objects, a vehicle including an arrangement for determining vehicle occupant position relative to a fixed structure within the vehicle comprises an array structured and arranged to receive an image of a portion of the passenger compartment of the vehicle in which the occupant is likely to be situated, a lens arranged between the array and the portion of the passenger compartment, adjustment means for changing the image received by the array, and processor means coupled to the array and the adjustment means. The processor means determine, upon changing by the adjustment means of the image received by the array, when the image is clearest whereby a distance between the occupant and the fixed structure is obtainable based on the determination by the processor means when the image is clearest. The image may be changed by adjusting the lens, e.g., adjusting the focal length of the lens and/or the position of the lens relative to the array, by adjusting the array, e.g., the position of the array relative to the lens, and/or by using software to perform a focusing process.
The array may be arranged in several advantageous locations on the vehicle, e.g., on an A-pillar of the vehicle, above a top surface of an instrument panel of the vehicle and on an instrument panel of the vehicle and oriented to receive an image reflected by a windshield of the vehicle. The array may be a CCD array with an optical liquid crystal or electrochromic glass filter coupled to the array for filtering the image of the portion of the passenger compartment. The array could also be a CMOS array.
In a preferred embodiment, the processor means are coupled to an occupant protection device and control the occupant protection device based on the distance between the occupant and the fixed structure. For example, the occupant protection device could be an airbag whereby deployment of the airbag is controlled by the processor means. The processor means may be any type of data processing unit such as a microprocessor.
This arrangement could be adapted for determining distance between the vehicle and exterior objects, in particular, objects in a blind spot of the driver. In this case, such an arrangement would comprise an array structure and arranged to receive an image of an exterior environment surrounding the vehicle containing at least one object, a lens arranged between the array and the exterior environment, adjustment means for changing the image received by the array, and processor means coupled to the array and the adjustment means. The processor means determine, upon changing by the adjustment means of the image received by the array, when the image is clearest whereby a distance between the object and the vehicle is obtainable based on the determination by the processor means when the image is clearest. As before, the image may be changed by adjusting the lens, e.g., adjusting the focal length of the lens and/or the position of the lens relative to the array, by adjusting the array, e.g., the position of the array relative to the lens, and/or by using software to perform a focusing process. The array may be a CCD array with an optional liquid crystal or electrochromic glass filter coupled to the array for filtering the image of the portion of the passenger compartment. The array could also be a CMOS array. In a preferred embodiment, the processor means are coupled to an occupant protection device and control the occupant protection device based on the distance between the occupant and the fixed structure. For example, the occupant protection device could be an airbag whereby deployment of the airbag is controlled by the processor means. The processor means may be any type of data processing unit such as a microprocessor.
Also, at least one of the above-listed objects is achieved by an arrangement for determining vehicle occupant presence, type and/or position relative to a fixed structure within the vehicle, the vehicle having a front seat and an A-pillar. The arrangement comprises a first array mounted on the A-pillar of the vehicle and arranged to receive an image of a portion of the passenger compartment in which the occupant is likely to be situated, and processor means coupled to the first array for determining the presence, type and/or position of the vehicle occupant based on the image of the portion of the passenger compartment received by the first array. The processor means preferably are arranged to utilize a pattern recognition technique, e.g., a trained neural network, sensor fusion, fuzzy logic. The processor means can determine the vehicle occupant presence, type and/or position based on the image of the portion of the passenger compartment received by the first array.
In some embodiments, a second array is arranged to receive an image of at least a part of the same portion of the passenger compartment as the first array. The processor means are coupled to the second array and determine the vehicle occupant presence, type and/or position based on the images of the portion of the passenger compartment received by the first and second arrays. The second array may be arranged at a central portion of a headliner of the vehicle between sides of the vehicle.
The determination of the occupant presence, type and/or position can be used in conjunction with a reactive component, system or subsystem so that the processor means control the reactive component, system or subsystem based on the determination of the occupant presence, type and/or position. For example, if the reactive component system or subsystem is an airbag assembly including at least one airbag, the processor means control one or more deployment parameters of the airbag(s).
The arrays may be CCD arrays with an optional liquid crystal or electrochromic glass filter coupled to the array for filtering the image of the portion of the passenger compartment. The arrays could also be CMOS arrays, active pixel cameras and HDRC cameras.
Another embodiment of the invention is an arrangement for obtaining information about a vehicle occupant within the vehicle which comprises transmission means for transmitting a structured pattern of light, e.g., polarized light, into a portion of the passenger compartment in which the occupant is likely to be situated, an array arranged to receive an image of the portion of the passenger compartment, and processor means coupled to the array for analyzing the image of the portion of the passenger compartment to obtain information about the occupant. The transmission means and array are proximate one another and the information obtained about the occupant is a distance from the location of the transmission means and the array. The processor means obtain the information about the occupant utilizing a pattern recognition technique. The information about the occupant can be used in conjunction with a reactive component, system or subsystem so that the processor means control the reactive component, system or subsystem based on the determination of the occupant presence, type and/or position. For example, if the reactive component, system or subsystem is an airbag assembly including at least one airbag, the processor means control one or more deployment parameters of the airbag(s).